Device and method for displaying augmented reality

ABSTRACT

Provided is a device for displaying augmented reality including an optical engine configured to output light of a virtual image, a waveguide configured to output light of the virtual image and transmit light of a real scene, a first lens part and a second lens part, and a processor, the first lens part being configured to tune a focus of the virtual image and including a first focus-tunable lens having a refractive power tunable by the processor and a fixed refractive lens having a refractive power, the second lens part being configured to compensate distortion of the real scene and including a second focus-tunable lens having a refractive power that is tunable by the processor, and the processor being further configured to determine the first refractive power based on vision information of the user, attribute depth information of the virtual image, and fixed refractive power of the fixed refractive lens.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2020-0067317, filed on Jun. 3, 2020and Korean Patent Application No. 10-2020-0124748, filed on Sep. 25,2020, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated by reference herein in their entireties.

BACKGROUND 1. Field

The disclosure relates to a device and method for displaying augmentedreality (AR), and more particularly, to a device for displaying AR,which includes a focus-tunable lens, and a method of displaying AR.

2. Description of Related Art

An augmented reality (AR) device enables a user to see AR, and mayinclude, for example, AR glasses. An image optical system of the ARdevice may include an image generation device that generates an imageand a waveguide that guides the generated image to eyes of a user.

An image output from the image generation device, for example, aprojector, etc. may be radiated to the eyes through the waveguide,whereby a user may observe the image. In a display using the waveguide,a focal distance of a virtual image may be, for example, infinite, andthus, for an immersive AR environment, a means for positioning a focaldistance of a virtual image to be an arbitrary distance where a realobject is located is needed. Meanwhile, among users using an AR device,a user whose vision is corrected with glasses needs to use an additionalmeans such as an optical clip. However, due to the inconvenience of theoptical clip, an AR device having a vision correction function forpeople with low vision by using a focus-tunable lens is being studied.

SUMMARY

The disclosure provides an AR device configured to perform self-visioncorrection.

The disclosure also provides an immersive AR environment.

The disclosure further provides an AR environment in which a quality ofa virtual image is improved.

Technical problems to be solved are not limited to the technicalproblems described above, and other technical problems may exist.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

According to an aspect of an example embodiment, there is provided adevice for displaying augmented reality (AR), the device including anoptical engine configured to output light of a virtual image, awaveguide configured to output the light of the virtual image receivedfrom the optical engine and transmit light of a real scene, a first lenspart provided on a first surface of the waveguide, a second lens partprovided on a second surface of the waveguide opposite to the firstsurface, and a processor, wherein the first lens part is configured totune a focus of the virtual image and correct a user's vision, the firstlens part including a first focus-tunable lens having a first refractivepower that is tunable by the processor and a fixed refractive lenshaving a fixed refractive power, wherein the second lens part isconfigured to compensate distortion of the real scene caused by thefirst lens part, and the second lens part including a secondfocus-tunable lens having a second refractive power that is tunable bythe processor, and wherein the processor is further configured todetermine the first refractive power of the first focus-tunable lensbased on vision information of the user, attribute depth information ofthe virtual image, and fixed refractive power information of the fixedrefractive lens.

The first refractive power of the first focus-tunable lens may satisfy

${D_{1} = {{- D_{fixed}} + D_{correction} - \frac{1}{f}}},$

where D₁ indicates the first refractive power of the first focus-tunablelens, D_(fixed) indicates the fixed refractive power of the fixedrefractive lens, D_(correction) indicates a correction-requiredrefractive power for correcting ametropia of the user, and f indicates afocal distance of the virtual image.

The device may further include a memory configured to store the fixedrefractive power D_(fixed) of the fixed refractive lens, thecorrection-required refractive power D_(correction) of the user, and thefocal distance f of the virtual image, wherein the processor is furtherconfigured to read the fixed refractive power of the fixed refractivelens, the correction-required refractive power of the user, and focaldistance of the virtual image from the memory and obtain the firstrefractive power D₁ of the first focus-tunable lens as

${- D_{fixed}} + D_{correction} - {\frac{1}{f}.}$

The device may further include a memory configured to store the focaldistance f of the virtual image and a modified correction-requiredrefractive power D_(modified) in which the fixed refractive power of thefixed refractive lens is reflected, wherein the modifiedcorrection-required refractive power D_(modified) satisfiesD_(modified)=−D_(fixed)+D_(correction), and wherein the processor isfurther configured to read the modified correction-required refractivepower and the focal distance of the virtual image from the memory andobtain the first refractive power D₁ of the first focus-tunable lens as

$D_{modified} - {\frac{1}{f}.}$

A second refractive power D₂ of the second focus-tunable lens maysatisfy

$D_{2} = {\frac{1}{f}.}$

The fixed refractive lens may be a concave lens having a negative (−)refractive power.

The first focus-tunable lens and the second focus-tunable lens may beliquid crystal lenses.

The second focus-tunable lens may be provided between the waveguide andthe fixed refractive lens, and wherein the first focus-tunable lens, thewaveguide, and the second focus-tunable lens may have a stack structure.

The device may further include a user input interface configured toreceive at least any one of the vision information of the user or thefocal distance of the virtual image based on a user input.

The first lens part may further include a polarization plate provided onan incident surface of the fixed refractive lens or an emission surfaceof the fixed refractive lens.

The second lens part may further include a second fixed refractive lensconfigured to compensate distortion of the real scene caused by thefirst lens part and the second focus-tunable lens.

The second fixed refractive lens may be a convex lens having a positive(+) refractive power.

The second refractive power D₂ of the second focus-tunable lens maysatisfy

${D_{2} = {\frac{1}{f} - D_{{fixed}\; 2}}},$

where D_(fixed2) indicates a fixed refractive power of the second fixedrefractive lens.

The device may further include a gaze tracking sensor configured toobtain gaze information of the user.

The processor may be further configured to obtain a gaze point from thegaze information of the user obtained by the gaze tracking sensor, anddetermine the focal distance of the virtual image based on the obtainedgaze point.

The processor may be further configured to control the optical engine tooutput at least one first character of a preset size, obtain at leastone first user input with respect to the at least one first character,compare the at least one first character with the at least one firstuser input, determine the first refractive power of the firstfocus-tunable lens based on a result of the comparing, and determine thecorrection-required refractive power of the user based on the determinedfirst refractive power of the first focus-tunable lens.

The at least one first character and at least one second character mayhave sizes corresponding to preset corrected vision, and the at leastone first character and the at least one second character are displayedto a preset depth for vision measurement of the user.

The device may be a glasses-type device.

According to an aspect of another example embodiment, there is provideda method of displaying augmented reality (AR) in an AR device thatincludes an optical engine configured to output light of a virtual imageand a waveguide configured to output the light of the virtual image andtransmit light of a real scene, the method including providing a firstlens part including a fixed refractive lens and a first focus-tunablelens and a second lens part including a second focus-tunable lens onopposite surfaces of the waveguide, obtaining a first refractive powerof the first focus-tunable lens based on vision information of a user,focal distance of the virtual image, and a fixed refractive power of thefixed refractive lens, and obtaining a second refractive power of thesecond focus-tunable lens to compensate for distortion of the real scenecaused by the first lens part.

The obtaining of the first refractive power of the first focus-tunablelens may include reading a fixed refractive power D_(fixed) of the fixedrefractive lens, a correction-required refractive power of the user, andthe focal distance f of the virtual image from a memory, and obtaining afirst refractive power D₁ of the first focus-tunable lens satisfying

${D\; 1} = {{- D_{fixed}} + D_{correction} - {\frac{1}{f}.}}$

According to an aspect of another example embodiment, there is provideda device for displaying augmented reality (AR), the device including anoptical engine configured to output light of a virtual image, awaveguide configured to output the light of the virtual image receivedfrom the optical engine and transmit light of a real scene, a first lenspart provided on a first surface of the waveguide, a second lens partprovided on a second surface of the waveguide opposite to the firstsurface, a microphone configured to receive a voice input of the user,and a processor, wherein the first lens part is configured to tune afocus of the virtual image and correct a user's vision, the first lenspart including a first focus-tunable lens having a first refractivepower that is tunable by the processor and a fixed refractive lenshaving a fixed refractive power, wherein the second lens part isconfigured to compensate distortion of the real scene caused by thefirst lens part, and the second lens part including a secondfocus-tunable lens having a second refractive power that is tunable bythe processor, and wherein the processor is further configured todetermine the first refractive power of the first focus-tunable lensbased on vision information of the user, attribute depth information ofthe virtual image, and fixed refractive power information of the fixedrefractive lens.

The processor may be further configured to control the optical engine tooutput at least one first character of a preset size, obtain at leastone first voice input received by the microphone with respect to the atleast one first character, compare the at least one first character withthe at least one first voice input, determine the first refractive powerof the first focus-tunable lens based on a result of the comparing, anddetermine a correction-required refractive power of the user based onthe determined first refractive power of the first focus-tunable lens.

The at least one first character may have a size corresponding to presetcorrected vision, and the at least one first character may be displayedto a preset depth for vision measurement of the user.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates an exterior of an augmented reality (AR) deviceaccording to an embodiment;

FIG. 2 is a plan view illustrating the AR device of FIG. 1;

FIG. 3 illustrates arrangement of an optical engine and optical partsaccording to an embodiment;

FIG. 4 is a block diagram of the AR device of FIG. 1;

FIG. 5 illustrates a first focus-tunable lens according to anembodiment;

FIG. 6 is a phase profile of a first focus-tunable lens when a controlsignal has a voltage profile corresponding to a concave lens having acertain refractive power;

FIG. 7 is a phase profile of a first focus-tunable lens when a controlsignal has a voltage profile corresponding to a concave lens having acertain refractive power;

FIG. 8 is a flowchart for describing an operation of an AR deviceaccording to an embodiment;

FIG. 9 is a flowchart for describing an operation of an AR deviceaccording to an embodiment;

FIG. 10 is a view for describing an operation of an AR device accordingto an embodiment;

FIG. 11 illustrates arrangement of optical parts of an AR deviceaccording to an embodiment;

FIG. 12 illustrates arrangement of optical parts of an AR deviceaccording to an embodiment;

FIG. 13 is a block diagram of an AR device according to an embodiment;

FIG. 14 illustrates a gaze tracking sensor according to an embodiment;

FIG. 15 illustrates a three-dimensional (3D) eyeball model with respectto a gaze direction of a user;

FIG. 16 is a view for describing a relationship between a gaze angle anda gaze point in a left eye and a right eye;

FIG. 17 is a view for describing a relationship between a gaze angle anda gaze point in an upward gaze direction;

FIG. 18 is a flowchart for describing an operation of an AR deviceaccording to an embodiment;

FIG. 19 is a view for describing an operation of an AR device accordingto an embodiment;

FIG. 20 is a block diagram of an AR device according to an embodiment;

FIG. 21 illustrates an example where an AR device according to anembodiment performs an operation to obtain a correction-requiredrefractive power of a user when a correct answer rate of a voice inputof the user is low;

FIG. 22 illustrates an example where an AR device according to anembodiment performs an operation to obtain the correction-requiredrefractive power of the user when the correct answer rate of the voiceinput of the user is normal;

FIG. 23 illustrates an example where an AR device according to anembodiment performs an operation to obtain the correction-requiredrefractive power of the user when the correct answer rate of the voiceinput of the user is high; and

FIG. 24 is a flowchart for describing an operation of an AR deviceaccording to an embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the disclosure will be described in detailwith reference to the attached drawings to allow those of ordinary skillin the art to easily carry out the embodiments of the disclosure.However, the disclosure may be implemented in various forms, and are notlimited to the embodiments of the disclosure described herein. Toclearly describe the disclosure, parts that are not associated with thedescription have been omitted from the drawings, and throughout thespecification, identical reference numerals refer to identical parts.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Although terms used in embodiments of the specification are selectedwith general terms popularly used at present under the consideration offunctions in the disclosure, the terms may vary according to theintention of those of ordinary skill in the art, judicial precedents, orintroduction of new technology. In addition, in a specific case, theapplicant voluntarily may select terms, and in this case, the meaning ofthe terms is disclosed in a corresponding description part of thedisclosure. Thus, the terms used in the specification should be definednot by the simple names of the terms but by the meaning of the terms andthe contents throughout the disclosure.

In the disclosure, augmented reality (AR) may be displaying a virtualimage by overlaying the virtual image on a physical environment space ora real object in a real world.

In the disclosure, an AR device may be a device capable of expressing‘AR’, and may include not only AR glasses in the form of glasses worn ona facial part of a user, but also a head-mounted display (HMD) or an ARhelmet, etc., worn on a head part of the user.

In the disclosure, a real scene may be a scene of the real world anobserver or the user sees through the AR device, and may include realworld object(s).

The virtual image may be an image generated through an optical engine.The virtual image may include both a static image and a dynamic image.The virtual image may be an image which is observed together with thereal scene and shows information regarding the real object in the realscene or information or a control menu, etc., regarding an operation ofthe AR device. The ‘virtual object’ may be expressed as a partial regionof the virtual image. The virtual object may indicate informationrelated to a real object. The virtual object may include at least oneof, for example, a character, a number, a sign, an icon, an image, oranimation.

In the disclosure, a focus-tunable lens may be a lens in which a focaldistance is tunable. As the focus-tunable lens, a liquid crystal (LC)lens, a liquid lens, or other well-known focus-tunable optical systemsmay be used. As described above, when the user sees the virtual image, adistance of the virtual image may be adjusted through the focus-tunablelens.

In the disclosure, a focus may be a point at which a straight lineextending from light parallel to an optical axis of a lens meets anoptical system after passing through the lens (or the optical system).On a principal plane of the lens (or the optical system), a distance tothe focus in the air may be a focal distance.

In the disclosure, a refractive index may be a rate at which the speedof light is reduced in a medium in comparison to a vacuum.

In the disclosure, a refractive power may be a force that changes adirection of light or an optical path by a curved surface of the lens.The unit of the refractive power is m⁻¹ or a diopter (D), a value ofwhich is expressed with a reciprocal number of a focal distance. Thediopter is referred to as a power of the lens having a correspondingrefractive power. The sign of the refractive power is positive (+) for aconvex lens and negative (−) for a concave lens.

In the disclosure, visual acuity (VA) may be the spatial resolvingability of eyes, i.e., the ability of the eyes to identify fine detailswhen a stationary object is seen with the eyes. Excessively high or lowametropia is a cause for myopia or hyperopia, which may be correctedwith a means such as glasses, contact lenses, vision correction surgery,or the like. A corrected vision may be a measured a vision of a userwearing a lens having a certain refractive power. A correction-requiredrefractive power means a refractive power required for achieving acorrected vision.

In the disclosure, a depth of a virtual image may be a distance or aposition in which the user recognizes existence of the virtual image ona space when the user sees the virtual image. A 3D image using binoculardisparity generates a left-eye virtual image and a right-eye virtualimage in different gaze directions, and in this case, the different gazedirections may include a gaze direction with the left eye of the userand a gaze direction from with right eye of the user. Thus, the depth ofthe virtual image in the 3D image using binocular disparity may be adistance converted from disparity (i.e., binocular disparity) based onthe gaze direction with the left eye and the gaze direction with theright eye.

In the disclosure, the gaze direction may be a direction in which theuser gazes, and the ‘gaze’ may be a virtual line directed from a pupilof the user in the gaze direction. The gaze direction is calculated frominformation obtained mainly in the gaze tracking sensor to estimate thegaze.

In the disclosure, the gaze point may be a point at which the usergazes, and may be calculated as a point at which the gazes of both eyesof the user intersect. When the user sees the 3D image using binoculardisparity, the user recognizes the 3D image based on the binoculardisparity, the gaze point obtained through a convergent angle of theeyes of the user may be a point in which the user recognizes existenceof the virtual object (i.e., the depth of the virtual image).

Hereinafter, the disclosure will be described with reference to theaccompanying drawings.

FIG. 1 illustrates an exterior of an AR device 100 according to anembodiment, and FIG. 2 is a plane view of the AR device 100 of FIG. 1.

Referring to FIGS. 1 and 2, the AR device 100 according to theembodiment may be AR glasses configured to be worn by the user and mayinclude a glasses-type body 101.

The glasses-type body 101 may include, for example, a frame 102 andtemples 103. The frame 102 in which glass lenses 104L and 104R arepositioned may have, for example, the shape of two rims connected by abridge. The glass lenses 104L and 104R are examples, and may have or maynot have a refractive power (a power). The glass lenses 104L and 104Rmay be formed integrally, and in this case, the rims of the frame 102may not be distinguished from the bridge. The glass lenses 104L and 104Rmay be omitted.

The temples 103 may be respectively connected to both end portions ofthe frame 102 and extend in a direction. The frame 102 and the temples103 may be connected by a hinge 105. The hinge 105 is an example, andthe glasses-type body 101 may include a member connecting the frame 102with the temples 103. In another example, the frame 102 and the temples103 may be connected integrally or continuously.

In the glasses-type body 101, an optical engine 110, a waveguide 120, afirst lens part 130, a second lens part 140, and electronic parts 190may be arranged.

The optical engine 110 may be configured to generate light of thevirtual image, and may be an optical engine of a projector, whichincludes an image panel, an illuminating optical system, a projectingoptical system, etc. The optical engine 110 may include a left-eyeoptical engine 110L and a right-eye optical engine 110R. The left-eyeoptical engine 110L and the right-eye optical engine 110R may bepositioned on both end portions of the frame 102. In another example,the left-eye optical engine 110L and the right-eye optical engine 110Rmay be respectively positioned in a left temple 103L and a right temple103R.

The optical parts may be configured to deliver light of the virtualimage generated in the optical engine 110 and light of a real scene tothe pupils of the user, and may include the waveguide 120, the firstlens part 130, and the second lens part 140. The optical parts may bearranged in the left side and the right side of the glasses-type body101. Left-eye optical parts and right-eye optical parts may be arrangedor attached in the left glass lens 104L and the right glass lens 104R.Alternatively, left-eye optical parts and right-eye optical parts may bemounted in the frame 102 separately from the glass lenses 104L and 104R.In another example, the left-eye optical parts and the right-eye opticalparts may be formed integrally and mounted on the frame 102. In anotherexample, the optical parts may be arranged in any one of the left sideand the right side of the glasses-type body 101.

The electronic parts 190 may include a processor (170 of FIG. 4), a userinput interface (150 of FIG. 4), and a memory (160 of FIG. 4), and maybe positioned in any one of the frame 102 or the temples 103 of theglasses-type body 101 or distributed in a plurality of positions, andmay be mounted on a printed circuit board (PCB), a flexible PCB (FPCB),etc. Although not shown, a first lens driving driver circuit that drivesa first focus-tunable lens 131 may be arranged adjacent to the firstfocus-tunable lens 131. A second lens driving driver circuit that drivesa second focus-tunable lens 141 may be arranged adjacent to a secondfocus-tunable lens 141. In another example, the first and second lensdriving driver circuits may be completely or partially positioned on,for example, a main board.

Referring to FIGS. 3 and 4, optical structure and operation of the ARdevice 100 according to an embodiment will be described in more detail.

FIG. 3 schematically illustrates the AR device 100 according to anembodiment, and FIG. 4 is a block diagram showing components of the ARdevice 100 according to an embodiment.

Referring to FIG. 3, the AR device 100 according to an embodiment may bean optical system configured to display both a virtual image and a realscene, and may include the optical engine 110, the waveguide 120, thefirst lens part 130, and the second lens part 140.

The optical engine 110 may be configured to output light L_(V) of avirtual image.

In an embodiment, the optical engine 110 may include a light source thatoutputs light, an image panel that forms a two-dimensional (2D) virtualimage by using the light output from the light source, and a projectingoptical system that projects the light L_(V) of the virtual image formedon the image panel, and may operate as, for example, a small projector.The light source may be, for example, a light-emitting diode (LED) or alaser diode (LD). The image panel may be, for example, a liquid crystalpanel, a liquid crystal on silicon (LCoS) panel, or a digitalmicromirror device (DMD) panel. The projecting optical system mayinclude at least one sheet of a projection lens.

In another embodiment, the optical engine 110 may include a light sourcethat outputs light and a two-axis scanner that two-dimensionally scansthe light output from the light source.

In another embodiment, the optical engine 110 may include a light sourcethat outputs light, a linear image panel that forms a linear image(i.e., a one-dimensional (1D) image) by using the light output from thelight source, and a one-axis scanner that scans light of the linearimage formed in the linear image panel.

The light L_(V) of the virtual image may be output from the waveguide120 and light L_(R) of the real scene may pass through the waveguide120. The waveguide 120 may be formed as a single layer or multiplelayers of a transparent material in which the light may propagate whilebeing internally reflected. Herein, the transparent material may be amaterial through which light in a visible light band passes. Atransparency of the transparent material may not be 100% and thetransparent material may have a certain color. The waveguide 120 mayhave the shape of a flat plate or a curved plate.

The waveguide 120 may include an input region to which the light L_(V)of the virtual image projected facing the optical engine 120 is input, apropagation region through which the incident light L_(V) of the virtualimage propagates, and an output region from which the light L_(V) of thevirtual image propagated from the propagation region is output. Theinput region and the output region are separated from each other. Thepropagation region may be positioned between the input region and theoutput region or may be positioned to overlap with at least a part ofthe input region or the output region.

In the input region, the propagation region, and the output region, aninput diffraction grating, a propagation diffraction grating, and anoutput diffraction grating are provided, respectively. When thewaveguide 120 includes a single layer, the input diffraction grating,the propagation diffraction grating, and the output diffraction gratingmay be formed on a surface of the waveguide 120 facing the opticalengine 110 and/or an opposite surface. When the waveguide 120 includesmultiple layers, the input diffraction grating, the propagationdiffraction grating, and the output diffraction grating may be formed oneach layer or some layers of the waveguide 120. The input diffractiongrating may be adapted to couple the light L_(V) output from the opticalengine 110 to the waveguide 120. The propagation diffraction grating maybe adapted to deliver the light L_(V) input from the input diffractiongrating to the output diffraction grating. For example, the propagationdiffraction grating may be an expansion grating that causes the inputlight L_(V) to be replicated into multiple ones. The expansion gratingmay be adapted to split the incident light L_(V) into a plurality ofbeamlets for propagation across the entire output region, when theincident light L_(V) is propagated through total reflection in thewaveguide 120. The output diffraction grating may be adapted to outputthe light L_(V) propagated in the waveguide 120 to the outside of thewaveguide 120 and may also operate as a propagation diffraction grating,for example, an expansion grating. A projection optical system of theoptical engine 110 may include a collimating lens and the light L_(V)emitted by the collimating lens may be parallel light, such that thelight L_(V) finally delivered to the eyes through the waveguide 120 maybe substantially regarded as a parallel pencil. For example, the lightL_(V) of the virtual image output through the output diffraction gratingmay be regarded as light substantially emitted from infinity. Herein,‘substantially’ may mean that the virtual image is sufficiently far,substantially close to infinity in terms of visual perspectiverecognized by a human.

The waveguide 120 may be mounted on a frame such that the output regionis positioned in front of the pupils of the user when the user wears theAR device 100. As the waveguide 120 is formed of a transparent material,the user may see the real scene as well as the virtual image through theAR device 100, and thus the AR device 100 may implement AR.

The first lens part 130 may perform focus tuning of the virtual imageand vision correction for the user, and thus may be positioned at a sideof the waveguide 120 from which the virtual image is output. When theuser wears the AR device 100, the first lens part 130 may be positionedbetween the waveguide 120 and the user's eyes.

The first lens part 130 may include the first focus-tunable lens 131 anda fixed refractive lens 133.

The first focus-tunable lens 131 may be a lens with a first refractivepower that varies with a control signal of a processor (170 of FIG. 4).For example, the first focus-tunable lens 131 may be a lens with a focaldistance that varies with the control signal of the processor 170.

The first focus-tunable lens 131 according to an embodiment may be aliquid crystal (LC) lens. For example, in the first focus-tunable lens131, liquid crystal may be positioned between upper and lowertransparent substrates, and a common electrode and lens electrodeshaving a certain pattern are arranged on a side where the upper andlower transparent substrates contact the liquid crystal. The commonelectrode and the lens electrodes with the certain pattern may betransparent electrodes. In the LC lens, a refractive index distributionof liquid crystal generated upon application of voltage between thecommon electrode and the lens electrodes may simulate a Fresnel lens.

FIG. 5 illustrates the first focus-tunable lens 131 according to anembodiment. Referring to FIG. 5, the first focus-tunable lens 131 has astructure in which an LC layer 1314 is interposed between a firstsubstrate 1311 and a second substrate 1318. A plurality of firstelectrodes 1312 having a certain pattern may be provided on the firstsubstrate 1311. The first electrodes (lens electrodes) 1312 may betwo-dimensionally arranged on the first substrate 1311. In anembodiment, the first electrodes 1312 may be formed of a concentric ringpattern. In another embodiment, the first electrodes 1312 may be formedas a two-dimensional (2D) pixel array pattern. Voltage may be applied tothe first electrodes 1312 individually (or independently) or in the unitof a certain group. A common second electrode (common electrode) 1317may be provided on the second substrate 1318. The second electrode 1317may be a reference electrode for the first electrodes 1312. Depending oncircumstances, the second electrode 1317 may also have the form of anelectrode array. The positions of the first electrodes 1312 and thesecond electrode 1317 may be interchanged with each other. The firstfocus-tunable lens 131 includes alignment layers 1313 and 1316 thatalign LC molecules 1315 in the LC layer 1314 in a certain direction. Theoriginal alignment of the LC molecules 1315 may be determined by adirection of a force applied to the alignment layers 1313 and 1316, butupon application of proper voltage, the LC molecules 1315 may rotate.Thus, when voltage is applied to the LC layer 1314, the refractive indexof the LC layer 1314 may change due to realignment of the LC molecules1315. As a valid refractive index is locally adjusted by spatiallyapplying a voltage profile to the first electrodes 1312 formed to havethe certain pattern, the LC layer 1314 may provide a phase modulationprofile having a certain focal distance.

FIG. 6 illustrates a phase profile of a first focus-tunable lens 131when a control signal is a voltage profile corresponding to a concavelens with a certain refractive power, for example, negative two diopter(−2D), and FIG. 7 illustrates a phase profile of the first focus-tunablelens 131 when the control signal is a voltage profile corresponding to aconcave lens with a certain refractive power, for example, negativethree diopter (−3D).

Referring to FIG. 6, upon application of a voltage profile correspondingto a concave lens with −2D between the first electrodes 1312 and thesecond electrode 1317, the refractive index distribution of thecorrespondingly generated LC layer 1314 simulates the Fresnel lenshaving a refractive power of −2D.

Referring to FIG. 7, upon application of a voltage profile correspondingto a concave lens with −3D between the first electrodes 1312 and thesecond electrode 1317, the refractive index distribution of thecorrespondingly generated LC layer 1314 simulates the Fresnel lenshaving a refractive power of −3D.

In the embodiment, the first focus-tunable lens 131 is an LC lens, forexample, but embodiments are not limited thereto. For example, anelectrooptic material having a refractive index changing with an appliedelectric field, such as electroactive polymers, liquid crystallinepolymers, or polymer dispersed liquid crystals, may be used in place ofLC. In another example, the first focus-tunable lens 131 may be a fluidlens that collects or disperses light by using an interfacial surfacebetween two types of liquid which are not mixed well.

In the LC layer or other focus-tunable lenses, a tunable range ortunable required time of a refractive power, a resolution, etc., may belimited according to a limitation of a manufacturing process,characteristics or driving scheme of an LC material, etc. In addition,the AR device 100 may be limited in terms of a mechanical size or apower in a sense that the AR device 100 is used worn by the user. Thus,as will be described later, there may be a limitation in solvingametropia of the user with the first focus-tunable lens 131.

The fixed refractive lens 133 may be an optical member having a fixedrefractive power. In an embodiment, the fixed refractive lens 133 may bea concave lens having a negative (−) refractive power. In theembodiment, the fixed refractive lens 133 is a concave lens, forexample, but embodiments are not limited thereto. In another example,the fixed refractive lens 133 may be a Fresnel lens, a graded refractiveindex (GRIN) lens, a meta lens, etc., with a negative (−) refractivepower. In another example, the fixed refractive lens 133 may be a convexlens having a positive (+) refractive power. Refractive powerinformation of the fixed refractive lens 133 may be stored in the memory160.

The second lens part 140 may compensate for distortion of the real scenecaused by the first lens part 130, and may be positioned on a surfaceopposite to a surface where the first lens part 130 is positioned, withthe waveguide 120 between the first lens part 130 and the second lenspart 140. That is, when the user wears the AR device 100, the secondlens part 140 may be arranged on the outer side of the waveguide 120 (aside in which the real scene is arranged).

The second lens part 140 may include the second focus-tunable lens 141.

The second focus-tunable lens 141 may be a lens with a second refractivepower that varies with the control signal of the processor 170. Thesecond focus-tunable lens 141 may have substantially the same structureas the first focus-tunable lens 131. In an embodiment, the secondfocus-tunable lens 141 may be an LC lens.

The first focus-tunable lens 131 and the second focus-tunable lens 141may be attached to the waveguide 120 to have a stack structure. Inanother example, the first focus-tunable lens 131 and the secondfocus-tunable lens 141 may be spaced by a certain distance from thewaveguide 120. The fixed refractive lens 133 may be attached to thefirst focus-tunable lens 131 or spaced by a certain distance from thefirst focus-tunable lens 131. In the embodiment, the first focus-tunablelens 131 is arranged between the waveguide 120 and the fixed refractivelens 133, for example, but embodiments are not limited thereto. Inanother example, the fixed refractive lens 133 may be arranged betweenthe waveguide 120 and the first focus-tunable lens 131.

Referring to FIG. 4, the AR device 100 may include the user inputinterface 150, the memory 160, and the processor 170, together with anoptical system including the first focus-tunable lens 131 and the secondfocus-tunable lens 141.

Among components shown in FIG. 4, a component having the same referencenumeral as that of a component shown in FIG. 3 is the same as thecomponent shown in FIG. 3. Thus, a repeated description will be omitted.

The user input interface 150 may be a means through which the userinputs data for controlling the AR device 100. For example, the userinput include 150 may include at least one of a keypad, a dome switch, atouch pad (a capacitive overlay type, a resistive overlay type, aninfrared beam type, a surface acoustic wave type, an integral straingauge type, a piezoelectric effect type, etc.), a jog wheel, a jogswitch, etc. The user input interface 150 may receive a user inputrelated to at least any one of the vision information of the user or thefocal distance of the virtual image.

The memory 160 may store various data, programs, or applications fordriving and controlling the AR device 100 and input/output signals ordata of a virtual image, under control of the processor 170. As anexample of various data for driving and controlling the AR device 100,user's vision information, refractive power information of a fixedrefractive lens, the refractive power tunable range of the first andsecond focus-tunable lenses 131 and 141, etc., may be stored in advancein the memory 160. A voltage profile for operating the first and secondfocus-tunable lenses 131 and 141 with corresponding refractive powersmay be stored in advance. Data of a virtual image may include attributedistance information of a virtual object in the virtual image.

The memory 160 may include at least one type of hardware devices among,for example, flash memory type, random access memory (RAM), staticrandom access memory (SRAM), read-only memory (ROM), electricallyerasable programmable ROM (EEPROM), programmable ROM (PROM), magneticmemory, a magnetic disc, and an optical disc.

The processor 170 may include, for example, at least one hardware amonga central processing unit (CPU), a microprocessor, a graphic processingunit (GPU), application specific integrated circuits (ASICs), digitalsignal processors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), or field programmable gate arrays(FPGAs), without being limited thereto.

The processor 170 may drive an operating system or an applicationprogram to control the overall operation of the AR device 100 includingthe optical engine 120 and the first and second focus-tunable lenses 131and 141, and perform various processing and operations with respect todata including image data. For example, the processor 170 may determinethe first refractive power of the first focus-tunable lens 131 based onuser's vision information, focal distance of the virtual image, andfixed refractive power information of the fixed refractive lens 133, andcontrol the first focus-tunable lens 131 with a control signalcorresponding to the first refractive power. For example, the controlsignal may be a voltage profile applied to the first focus-tunable lens131. In another example, the control signal may be a control commandsignal corresponding to preset voltage profiles. The processor 170 maydetermine the second refractive power of the second focus-tunable lens141 based on the focal distance of the virtual image, and control thesecond focus-tunable lens 141 with a control signal corresponding to thesecond refractive power.

FIG. 8 is a flowchart for describing an operation of the AR device 100according to an embodiment.

Referring to FIG. 8, to determine the first refractive power of thefirst focus-tunable lens 131 and the second refractive power of thesecond focus-tunable lens 141 by reflecting user's vision, the processor170 may load focal distance of the virtual image, the user's visioninformation, and the fixed refractive power information from the memory160, in operation S210.

As described above, the virtual image output from the waveguide 120 maybe regarded as being in a substantially infinite position. The user maysee the virtual image output from the waveguide 120 through the firstlens part 130, such that the focal position of the virtual image may bemoved by the first lens part 130.

When the user has ametropia, movement of the focal position of thevirtual image may be limited by the ametropia of the user and to correctthe ametropia, a correction refractive power is required. For example,when the user has myopia as ametropia, a correction lens for correctingmyopia may require a correction-required refractive power of minusdiopter (e.g., −3D). According to an embodiment, the user's visioninformation is a correction-required refractive power, and acorrection-required refractive power of the user may be stored in thememory 160. When there are a plurality of users, the user's visioninformation may include user identification information and thecorrection-required refractive power of the user. The user's visioninformation may be previously stored in the memory 160. In an embodimentof the disclosure, the user's vision information may be directly inputby the user through the user input interface 150. In an embodiment, theuser's vision information may be stored in another electronic device anddelivered from the other electronic device in a wired or wireless mannerand stored in the memory 160.

The virtual object in the virtual image may include at least one of, forexample, a character, a number, a sign, an icon, an image, or animation.The virtual object may be a 3D object as well as a 2D object.

In an embodiment, the virtual object may appear more natural to the userwhen the virtual object is recognized as being located at a certaindistance. For example, when the user sees a desk or a table, sitting ona chair or a sofa, the virtual image (the virtual object) may include animage of a product virtually placed on the desk or the table orinformation about a product placed on the desk or the table, and anattribute distance of the virtual image (the virtual object) may beabout 0.5 meter (m) to about 0.7 meter (m). In another example, when theuser does the shopping in a store, the virtual image (the virtualobject) may display information about a product at the store, and theattribute distance of the virtual image (the virtual object) may beabout 1 m to about 2 m. Thus, representative distance information of thevirtual image (the virtual object) or focal distance informationappropriate for an attribute of each virtual image (each virtual object)may be stored, together with virtual image data, in the memory 160.

Next, the processor 170 may determine the first refractive power of thefirst focus-tunable lens 131 based on the focal distance of the virtualimage, the vision information of the user, and the fixed refractivepower information of the fixed refractive lens 133, in operation S220.

As such, when the user has ametropia, the refractive power of the firstfocus-tunable lens 131 of the first lens part 130 may be defined asshown below in Equation 1.

$\begin{matrix}{D_{1} = {{- D_{fixed}} + D_{correction} - \frac{1}{f}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, D₁ may indicate the first refractive power of the firstfocus-tunable lens 131, D_(fixed) indicate a fixed refractive power ofthe fixed refractive lens 133, and D_(correction) may indicate acorrection-required refractive power for correcting ametropia of theuser. f indicates the focal distance of the virtual image.

The processor 170 may adjust the first refractive power of the firstfocus-tunable lens 131 such that the focal distance f of the virtualimage is the attribute distance of the virtual image, thereby enablingthe user to see the virtual image more naturally with the correctedvision.

In the embodiment, a case where there is the attribute distance of thevirtual image (the virtual object) is described as an example, but theattribute distance of the virtual image (the virtual object) may notexist. In another embodiment, the focal distance f of the virtual imagemay be a fixed value irrelevant to the attribute of the virtual image(the virtual object), and thus may be set to about 0.5 m or about 0.7 mbased on an aspect in which the AR device 100 is used. In anotherembodiment, the focal distance f of the virtual image may be a valueadjustable by a user's input, regardless of the attribute of the virtualimage (the virtual object).

Next, the processor 170 may determine the second refractive power of thesecond focus-tunable lens 141 based on the focal distance of the virtualimage, in operation S230.

The light departing from the real object may enter the pupils of theuser through the second lens part 140, the waveguide 120, and the firstlens part 130. Due to the first refractive power of the firstfocus-tunable lens 131 of the first lens part 130 and the fixedrefractive power of the first refractive lens 133, the light departingfrom the real scene may be refracted, causing distortion in the realscene. The second focus-tunable lens 141 of the second lens part 140 mayhave a certain refractive power to compensate for distortion in the realscene, caused by the first lens part 130. For example, the secondrefractive power of the second focus-tunable lens 141 may be determinedas shown below in Equation 2.

$\begin{matrix}{D_{2} = \frac{1}{f}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

Here, D₂ indicates the second refractive power of the secondfocus-tunable lens 141.

Table 1 shows the first refractive power of the first focus-tunable lens131, the second refractive power of the second focus-tunable lens 141,and the fixed refractive power of the fixed refractive lens 133 (theconcave lens) with respect to user's vision.

TABLE 1 Second Convex First Focus- Focus- Clas- Virtual Lens TunableLens (D₂) sifica- User's Vision Focus (D_(fixed)) (D₁) Tunable Lens tion(D_(correction)) (f) (−2D) (+3D~−3D) (+3D~−3D) 1 3D Myopia Virtual −2D  −3D   +2D (−3D) @ 0.5 m 2 2D Myopia −2D   −2D   +2D (−2D) 3 1.5DMyopia −2D −1.5D   +2D (−1.5D) 4 1D Myopia −2D   −1D   +2D (−1D) 5 3DMyopia Virtual −2D −2.5D +1.5D (−3D) @ 0.7 m 6 2D Myopia −2D −1.5D +1.5D(−2D) 7 1.5D Myopia −2D   −1D +1.5D (−1.5D) 8 1D Myopia −2D −0.5D +1.5D(−1D)

In Classification 1 of Table 1, when the correction-required refractivepower of the user is −3D, the focal distance (the virtual focus) of thevirtual image is 0.5 m, and the fixed refractive power of the concavelens is −2D, the first refractive power of the first focus-tunable lens131 may be −3D and the second refractive power of the secondfocus-tunable lens 141 may be +2D.

As another example, in Classification 5, when the correction-requiredrefractive power of the user is −3D, the focal distance (the virtualfocus) of the virtual image is 0.7 m, and the fixed refractive power ofthe concave lens is −2D, the first refractive power of the firstfocus-tunable lens 131 may be −2.5D and the second refractive power ofthe second focus-tunable lens 141 may be +1.5D. When the focal distanceof the virtual image is about 0.7 m,

$\frac{1}{f}$

may be generally regarded as 1.5D for calculation.

In general, a sum of the first refractive power of the firstfocus-tunable lens 131 of the first lens part 130 and the fixedrefractive power of the fixed refractive lens 133 may be asymmetric tothe second refractive power of the second focus-tunable lens 141. Thatis, an absolute value of a sum of refractive powers of the first lenspart 130 is not equal to an absolute value of the refractive power ofthe second lens part 140.

As described above, in the first and second focus-tunable lenses 131 and141, a tunable range or tunable required time of a refractive power, aresolution, etc., may be limited according to a limitation of amanufacturing process, characteristics or driving scheme of a material,etc. For example, the valid refractive power tunable range of the firstand second focus-tunable lenses 131 and 141 may be from about +3D toabout −3D. As shown in Table 1, the AR device 100 according to theembodiment may determine the first refractive power and the secondrefractive power of the first focus-tunable lens 131 and the secondfocus-tunable lens 141 within a valid refractive power tunable range.

The first focus-tunable lens 131 and the second focus-tunable lens 141may have a limitation in having high refractive power due to alimitation in pattern refinement of the lens electrode, for example, thefirst electrodes 1312 in FIG. 5 or instability of LC alignment at apoint requiring a rapid change in the phase of light. For example, whenthe user has ametropia, a required refractive power may be difficult tomanage merely with the first focus-tunable lens 131. In the embodiment,by arranging the fixed refractive lens (the concave lens) 133 in thefirst lens part 130, a refractive power load on the first focus-tunablelens 131 may be reduced, thereby achieving a high resolution of thevirtual image.

In the foregoing embodiment, a description is made of a case where theprocessor 170 loads the focal distance of the virtual image, the user'svision information, and the fixed refractive power information from thememory 160 and determine the first refractive power of the firstfocus-tunable lens 131, but embodiments are not limited thereto. For aparticular user, user's vision information and fixed refractive powerinformation are already fixed values, such that the user's visioninformation and the fixed refractive power information may be previouslycalculated as shown in Equation 3 provided below and previously storedin the memory 160.

D _(modified) =−D _(fixed) +D _(correction)  [Equation 3]

D_(modified) indicates a modified correction-required refractive power,and may be understood as a correction-required refractive power intowhich the refractive power of the fixed refractive lens is reflected.

FIG. 9 is a flowchart for describing an operation of an AR deviceaccording to an embodiment. The embodiment may correspond to a casewhere the modified correction-required refractive power D_(modified) ispreviously stored in the memory 160. Referring to FIG. 9, to determinethe first refractive power of the first focus-tunable lens 131 and thesecond refractive power of the second focus-tunable lens 141 byreflecting user's vision, the processor 170 may load the modifiedcorrection-required refractive power D_(modified) and the focal distancef of the virtual image from the memory 160, in operation S310.

Next, after the modified correction-required refractive powerD_(modified) and the focal distance f of the virtual image are read fromthe memory 160, the first refractive power of the first focus-tunablelens 131 of the first lens part 130 may be determined using Equation 4provided below in operation S320.

$\begin{matrix}{D_{1} = {D_{modified} - \frac{1}{f}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

As such, by using the modified correction-required refractive power inwhich the user's vision information and the fixed refractive powerinformation are previously calculated, the number of pieces ofinformation loaded from the memory 160 may be reduced and an operationfor determining the first refractive power may be further simplified.

Next, the processor 170 may determine the second refractive power of thesecond focus-tunable lens 141 based on the focal distance of the virtualimage as in Equation 2, in operation S330.

FIG. 10 is a flowchart for describing an operation of the AR device 100,according to an embodiment.

The user having ametropia may need a correction-required refractivepower due to the ametropia. A virtual image (a virtual object) O_(V)displayed on the AR device 100 may have a focal distance appropriate forattributes thereof. Alternatively, the virtual image (the virtualobject) O_(V) may use a representative distance previously input to theAR device 100 as a focal distance. When the user desires to see thevirtual image (the virtual object) O_(V), wearing the AR device 100, theprocessor 170 may determine the first refractive power of the firstfocus-tunable lens 131 and the second refractive power of the secondfocus-tunable lens 141 and control the first focus-tunable lens 131 andthe second focus-tunable lens 141 corresponding to the determined firstrefractive power and second refractive power, as described withreference to FIGS. 8 and 9. As a result, the user may correct visionusing the first lens part 130 in spite of having ametropia, and may beable to see the virtual image (the virtual object) O_(V) at the focaldistance f that is an infinite distance by using the first lens part130, such that the user may clearly and naturally see the virtual image(the virtual object) O_(V). In addition, the user may see the real scenewithout distortion caused by the first lens part 130, by using thesecond refractive power of the second lens part 140.

FIG. 11 illustrates arrangement of optical parts of an AR device 400according to an embodiment.

Referring to FIG. 11, the AR device 400 according to the embodiment mayinclude the optical engine 110, the waveguide 120, a first lens part430, and the second lens part 140. The AR device 400 according to theembodiment is the same as the above-described embodiments except thatthe first lens part 430 further includes a polarization plate 432, suchthat a description will be made based on a difference.

The first lens part 430 may include the first focus-tunable lens 131,the polarization plate 432, and the fixed refractive lens 133. In anembodiment, the polarization plate 432 may be arranged between the firstfocus-tunable lens 131 and the fixed refractive lens 133. Thepolarization plate 432 may pass first polarized light therethrough andblock second polarized light. The first polarized light may be linearpolarized light (e.g., p polarized light). The first focus-tunable lens131 may be an LC lens. A refractive index of the LC lens may vary withthe first polarized light (e.g., the p polarized light) and the secondpolarized light (e.g., s polarized light) that is orthogonal to thefirst polarized light due to the nature of double refraction. Thus, byarranging the polarization plate 432 between the first focus-tunablelens 131 and the fixed refractive lens 133, light (i.e., noise) having adifferent refraction magnitude among light passing through the firstfocus-tunable lens 131 may be canceled.

FIG. 11 shows that the polarization plate 432 is arranged between thefirst focus-tunable lens 131 and the fixed refractive lens 133, forexample, but embodiments are not limited thereto. In an embodiment, thepolarization plate 432 may be arranged between the waveguide 120 and thefirst focus-tunable lens 131. That is, the waveguide 120, thepolarization plate 432, the first focus-tunable lens 131, and the fixedrefractive lens 133 may be arranged in that order. In another example,the waveguide 120, the fixed refractive lens 133, the polarization plate432, and the first focus-tunable lens 131 may be arranged in that order,or the waveguide 120, the polarization plate 432, the fixed refractivelens 133, and the first focus-tunable lens 131 may be arranged in thatorder.

FIG. 12 illustrates arrangement of optical parts of an AR deviceaccording to an embodiment.

Referring to FIG. 12, an AR device 500 according to the embodiment mayinclude the optical engine 110, the waveguide 120, the first lens part130, and a second lens part 540. The first lens part 130 may include thefirst focus-tunable lens 131 and the first fixed refractive lens 133,and the second lens part 540 may include the second focus-tunable lens141 and a second fixed refractive lens 543. In an embodiment of thedisclosure, the second fixed refractive lens 543 may be a convex lenshaving a positive (+) refractive power. In the embodiment, the secondfixed refractive lens 543 is a convex lens, for example, but embodimentsare not limited thereto. In another example, the second fixed refractivelens 543 may be a Fresnel lens, a GRIN lens, a meta lens, etc., with apositive (+) refractive power. The AR device 500 according to theembodiment is the same as the above-described embodiments except thatthe second lens part 540 further includes the second fixed refractivelens 543.

A refractive power D′₂ the second lens part 540 has to have tocompensate for distortion of the real scene, caused by the first lenspart 130, may be determined by Equation 5 shown below.

$\begin{matrix}{D_{2}^{\prime} = \frac{1}{f}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

The refractive power D′₂ of the second lens part 540 is given as a sumof the second refractive power D₂ of the second focus-tunable lens 141and a fixed refractive power D_(fixed2) of the second fixed refractivelens 543, such that the refractive power D₂ of the second focus-tunablelens 141 is determined by Equation 6 shown below.

$\begin{matrix}{D_{2} = {\frac{1}{f} - D_{{fixed}\; 2}}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

As described above, the second lens part 540 compensates for distortionof the real scene caused by the first lens part 130, and the secondrefractive power to be managed by the second focus-tunable lens 131 ofthe second lens part 540 may be excessively high according to the user'svision, etc. In the embodiment, by distributing the second refractivepower required for the second focus-tunable lens 141 with the secondfixed refractive lens 543, a load on the second focus-tunable lens 141may be reduced, thereby achieving a high resolution of the virtualimage.

Table 2 shows the first refractive power D₁ of the first focus-tunablelens 131, the second refractive power D₂ of the second focus-tunablelens 141, the fixed refractive power D_(fixed1) of the first fixedrefractive lens 133 (the concave lens), and the fixed refractive powerD_(fixed2) of the second fixed refractive lens 543 (the convex lens)with respect to user's vision.

TABLE 2 Second First Fixed Second First Fixed Focus- Refractive Focus-User's Virtual Refractive Tunable Lens Tunable Vision Focus Lens(D_(fixed1)) Lens (D₁) (D_(fixed2)) Lens (D₂) Classification(D_(correction)) (f) (−2D) (+3D~−3D) (+1D) (+3D~−3D) 1 3D Myopia Virtual−2D   −3D +1D   +1D (−3D) @ 2 2D Myopia 0.5 m −2D   −2D +1D   +1D (−2D)3 1.5D −2D −1.5D +1D   +1D Myopia (−1.5D) 4 1D Myopia −2D   −1D +1D  +1D (−1D) 5 3D Myopia Virtual −2D −2.5D +1D +0.5D (−3D) @ 6 2D Myopia0.7 m −2D −1.5D +1D +0.5D (−2D) 7 1.5D −2D   −1D +1D +0.5D Myopia(−1.5D) 8 1D Myopia −2D −0.5D +1D +0.5D (−1D)

In Classification 1 of Table 2, when the user's correction-requiredrefractive power D_(correction) is −3D, the focal distance (a virtualfocus) f of the virtual image is 0.5 m, the fixed refractive powerD_(fixed1) of the first fixed refractive lens 133 (the concave lens) is−2D, and the fixed refractive power D_(fixed2) of the second fixedrefractive lens 543 (the convex lens) is +1D, the first refractive powerD₁ of the first focus-tunable lens 131 may be −3D and the secondrefractive power D₂ of the second focus-tunable lens 141 may be +1D.

In another example, in Classification 5, when the user'scorrection-required refractive power D_(correction) is −3D, the focaldistance (the virtual focus) f of the virtual image is 0.7 m, the fixedrefractive power D_(fixed1) of the first fixed refractive lens 133 (theconcave lens) is −2D, and the fixed refractive power D_(fixed2) of thesecond fixed refractive lens 543 (the convex lens) is +1D, the firstrefractive power D₁ of the first focus-tunable lens 131 may be −2.5D andthe second refractive power D₂ of the second focus-tunable lens 141 maybe +0.5D. When the focal distance of the virtual image is about 0.7 m,1/f may be generally regarded as 1.5D for calculation.

In general, a sum of the first refractive power of the firstfocus-tunable lens 131 of the first lens part 130 and the fixedrefractive power of the first fixed refractive lens 133 may beasymmetric to a sum of the second refractive power of the secondfocus-tunable lens 141 of the second lens part 540 and the fixedrefractive power of the second fixed refractive lens 543. That is, anabsolute value of a sum of refractive powers of the first lens part 130is not equal to an absolute value of a sum of refractive powers of thesecond lens part 540.

FIG. 13 is a block diagram of an AR device 600 according to anembodiment.

Referring to FIG. 13, the AR device 600 according to the embodiment mayinclude the user input interface 150, the memory 160, the processor 170,and a gaze tracking sensor 680, together with an optical systemincluding the optical engine 110, the first focus-tunable lens 131, andthe second focus-tunable lens 141. The AR device 600 according to theembodiment is substantially the same as the AR device 100 according tothe embodiment described with reference to FIG. 4 except that the ARdevice 600 further includes the gaze tracking sensor 680, such that adescription will be made of a difference occurring due to additionalinclusion of the gaze tracking sensor 680.

The gaze tracking sensor 680, which is a device for tracking a gazedirection of the eyes of the user, may detect an image of pupils of thehuman or detect a direction or a quantity in which illumination lightsuch as near-infrared light is reflected from the cornea, therebydetecting the gaze direction of the user. The gaze tracking sensor 680may include a left-eye gaze tracking sensor and a right-eye gazetracking sensor which detect the gaze direction of the left eye of theuser and the gaze direction of the right eye of the user, respectively.Detection of the gaze direction of the user may include obtaining gazeinformation related to the gaze of the user.

FIG. 14 illustrates the gaze tracking sensor 680 according to anembodiment. Referring to FIG. 14, the gaze tracking sensor 680 mayinclude an infrared radiator 681 and a plurality of infrared detectors685 a through 685 f. While six infrared detectors 685 a through 685 fare illustrated in FIG. 14, this is merely for convenience of adescription, and the number of plural infrared detectors 685 a through685 f is not limited to the illustration.

The infrared radiator 681 may radiate infrared light to a cornea part inwhich a crystalline lens of an eye E is located, and the plurality ofinfrared detectors 685 a through 685 f may detect the infrared lightreflected from the cornea. In an embodiment, the gaze tracking sensor680 may obtain information about the quantity of infrared light detectedby each of the plurality of infrared detectors 685 a through 685 f andobtain information about a gaze direction in which the eye E of the usergazes based on the obtained information about the quantity of theinfrared light. The gaze tracking sensor 680 may provide the obtainedinformation about the gaze direction to the processor 170. For example,the information about the gaze direction obtained by the gaze trackingsensor 680 may include gaze angle information in horizontal and verticaldirections of the left eye and gaze angle information in the horizontaland vertical directions of the right eye.

As the gaze tracking sensor 680 according to the embodiment, an IRscanner scheme using infrared illumination light is described forexample, but embodiments are not limited thereto. In another example,the gaze tracking sensor 680 may include an image sensor that capturesan image of the pupil of the human. Based on the captured image of theeye of the user, gaze angle information in the horizontal and verticaldirections of the left eye and gaze angle information in the horizontaland vertical directions of the right eye may be detected.

Referring back to FIG. 13, according to an embodiment, the gaze trackingsensor 680 may sense the eye of the user wearing the AR device 600 atcertain time intervals. The processor 170 may calculate the gaze pointof the user based on the information about the gaze directions of theleft eye and the right eye, detected by the gaze tracking sensor 680.For example, when the user sees an object of the real scene togetherwith a virtual image displayed by the AR device 100, the processor 170may determine a depth (i.e., a focal distance) of the virtual imagebased on the calculated gaze point.

Next, a method of calculating a gaze point from information about a gazedirection, measured by the gaze tracking sensor 680, will be describedwith reference to FIGS. 15 through 17.

FIG. 15 illustrates a three-dimensional (3D) eyeball model with respectto a gaze direction of a user.

Referring to FIG. 15, tracking of a gaze direction according to anembodiment may be performed based on a 3D eyeball model with respect toa gaze. Assuming that the 3D eyeball model with respect to the gaze is acomplete sphere and the eyeball ideally spatially rotates along thegaze, the gaze may be mathematically modeled as shown in Equation 7provided below:

$\begin{matrix}{{x = {{d \cdot \tan}\;\alpha}}{y = {{d \cdot \sec}\;{\alpha \cdot \tan}\;\beta}}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack \\{{\beta = {\sin^{- 1}\frac{\Delta\; y}{r}}}{\alpha = {\sin^{- 1}\frac{\Delta\; x}{r\;\cos\;\beta}}}} & \left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack\end{matrix}$

In Equation 7, d indicates a distance between a center Eo of the eye(eyeball) E of the user and a virtual screen S, α indicates an angle bywhich the eye of the user rotates in an x-axis (horizontal-axis)direction with respect to a case where the user's eye frontally gazes atthe virtual screen S, and β indicates an angle by which the eye of theuser rotates in a y-axis (vertical-axis) direction with respect to thecase where the user's eye frontally gazes at the virtual screen S. Inaddition, in Equation 8, r indicates a radius of a sphere assuming thatthe eye of the user is the sphere.

The eye tracking sensor 680 according to an embodiment of the disclosuremay measure a degree of rotation (e.g., α and β) of the eye (eyeball) Eof the user, and the AR device 500 may calculate two-dimensional (2D)position coordinates (x, y) of the gaze direction of the eye (eyeball) Eof the user on the virtual screen S by using the degree of rotation (αand β) of the eye (eyeball) E of the user. The degree of rotation (α andβ) of the eye (eyeball) E may be understood as gaze angle information inthe horizontal and vertical directions.

Actual movement of the eye may not include ideal 3D rotation, and inparticular, relaxation/contraction of eye muscles act greatly in termsof left/right gazes, such that an error may occur in estimation oftop/bottom gazes with respect to the left/right gazes based on an ideal3D rotation eyeball model. The AR device 600 may solve the error bycausing the user to see a random point and comparing a gaze directionestimated by the gaze tracking sensor 680 with an actual gaze directionwith respect to the point to statistically process them, therebyimproving accuracy.

FIG. 16 is a view for describing a relationship between a gaze angle anda gaze point in a left eye and a right eye, and FIG. 17 is a view fordescribing a relationship between a gaze angle and a gaze point in anupward gaze direction.

Referring to FIGS. 16 and 17, a focal distance may be estimated based ona difference between gaze directions (or gaze coordinates) of both eyesobtained through the gaze tracking sensor 680. When the focal distanceto the gaze point is obtained, gaze axes of the both eyes may not meeteach other, and in this case, a vertical-axis (y-axis) coordinate may becalculated as an average of vertical-axis (y-axis) coordinates of thetwo eyes assuming that the two eyes are in the same height. A distance abetween the both eyes may be assumed to be, for example, about 7 cm. Byusing a proportional expression based on a geometric(al) assumption, thefollowing Equation 9 may be obtained.

$\begin{matrix}{\frac{- z}{\Delta\; x} = \frac{d - z}{a}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack\end{matrix}$

In Equation 9, a distance d to a virtual screen and the distance abetween the eyes are required, and the distance d may be obtained bymeasuring a rotation angle of the eyeball using a gaze image in whichthe user gazes at the front. As a result, a distance D to the gaze pointmay be given by Equation 10 below.

$\begin{matrix}{D = {{d + z} = {{d + \frac{\Delta\;{xd}}{{\Delta\; x} - a}} = {\left( {1 + \frac{\Delta\; x}{{\Delta\; x} - a}} \right) \cdot d}}}} & \left\lbrack {{Equation}\mspace{14mu} 10} \right\rbrack\end{matrix}$

Δx indicates a horizontal interval between gaze coordinates of the botheyes on the virtual screen S, and may be obtained from gaze angles ofthe left eye and the right eye of the user as can be seen from Equations7 and 8.

FIG. 18 is a flowchart for describing an operation of the AR device 600according to an embodiment.

Referring to FIG. 18, to determine the first refractive power of thefirst focus-tunable lens 131 and the second refractive power of thesecond focus-tunable lens 141 by reflecting user's vision, the AR device600 may obtain the focal distance of the virtual image.

First, the gaze tracking sensor 680 of the AR device 600 may obtaininformation about the gaze direction of the left eye of the user andinformation about the gaze direction of the right eye of the user, inoperation S710.

Next, as the example described with reference to FIGS. 15 through 17,the processor 170 of the AR device 600 may obtain calculate a gaze pointfrom the information about the gaze direction of the left eye of theuser and the information about the gaze direction of the right eye ofthe user, in operation S720.

Next, the processor 170 may determine the focal distance of the virtualimage based on the obtained gaze point, in operation S730.

In an embodiment, when the user sees the real scene together with thevirtual image displayed by the AR device 100, the user may gaze at thereal object, which is a subject of interest of the user, in the realscene, and it may be natural that the virtual image (the virtual object)is placed in the same depth as the real object. Thus, a depth that issimilar to a depth to the gaze point of the user (i.e., a distancebetween the eye of the user and the gaze point) may be set to the focaldistance of the virtual image. Herein, the similar depth may include notonly a case where the focal distance of the virtual image is equal tothe depth to the gaze point of the user, but also a depth in a rangenaturally recognized by the user. For example, the focal distance of thevirtual image may be changed in an approximate size range of the realobject.

As described below, by setting the focal distance of the virtual imageto a depth corresponding to the gaze point of the user detected by thegaze tracking sensor 680, the user may naturally see the real scenetogether with the virtual image displayed by the AR device 100.

While operations S720 and S730 are described separately in theembodiment, operation S730 may be substantially omitted by regarding thecalculated distance to the gaze point as the focal distance.

The processor 170 may load the user's gaze information and the fixedrefractive power information from the memory 160, in operation S740.Operation S740 may be performed reversely to or simultaneously withoperations S710 through S730.

Next, the processor 170 may determine the first refractive power of thefirst focus-tunable lens 131 based on the focal distance of the virtualimage, the vision information of the user, and the fixed refractivepower information of the fixed refractive lens 133, in operation S720.For example, the first refractive power of the first focus-tunable lens131 of the first lens part 130 may be determined using Equation 1described above, and enables the user to naturally see the virtual imagewith corrected vision.

In the embodiment, a description is made of a case where the user'svision information and the fixed refractive power information arepreviously stored, as an example, but embodiments are not limitedthereto. As in the example described with reference to FIG. 9, theuser's vision information and the fixed refractive power information maybe previously calculated and stored as the modified correction-requiredrefractive power in the memory 160, and the processor 170 may determinethe first refractive power of the first focus-tunable lens 131 based onthe focal distance of the virtual image and the modifiedcorrection-required refractive power information.

Next, the processor 170 may determine the second refractive power of thesecond focus-tunable lens 141 of the second lens part 140, based on thefocal distance of the virtual image, in operation S760. The secondrefractive power of the second focus-tunable lens 141 may be determinedusing Equation 2 described above, and distortion of the real scene,caused by the first lens part 130, may be compensated.

In the embodiment, the virtual image (the virtual object) may be 3D aswell as 2D. For example, the virtual image may provide a cubic effectbased on binocular disparity. The virtual image using binoculardisparity may generate a left-eye virtual image and a right-eye virtualimage in different viewpoints, and in this case, the differentviewpoints may include a view point with the left eye of the user and aview point from with right eye of the user. Thus, by causing the virtualimage to have binocular disparity corresponding to the focal distancedetermined in operation S730, the user may see the virtual imagenaturally.

FIG. 19 is a view for describing an operation of an AR device accordingto an embodiment.

When the user wearing the AR device 100 sees a product (a real object)O_(R) in a store, the AR device 100 may display information about theproduct through a virtual image (a virtual object) O_(V). For example,when the user gazes at the product (the real object) O_(R), the gazetracking sensor 680 of the AR device 100 may track the gaze of the userand the processor 170 may calculate a gaze point from information abouta tracked gaze direction of the user and determine a distance to theproduct (the real object) O_(R) from the gaze point as the focaldistance f of the virtual image (the virtual object) O_(V). Theprocessor 170 may determine the first refractive power of the firstfocus-tunable lens 131 based on the focal distance f of the virtualimage (the virtual object) O_(V), the user's vision information, and thefixed refractive power information of the fixed refractive lens 133,determine the second refractive power of the second focus-tunable lens141 of the second lens part 140 based on the focal distance f of thevirtual image (the virtual object) O_(V), and control the first andsecond focus-tunable lenses 131 and 141 corresponding to the determinedfirst and second refractive powers. As a result, the user may correctvision using the first lens part 130 in spite of having ametropia, andmay draw the focal distance f from the infinite distance closely to theposition where the product (the real object) O_(R) is located, such thatthe user may more clearly and naturally see the virtual image (thevirtual object) O_(V). In addition, the user may see the product (thereal object) O_(R) without distortion caused by the first lens part 130,by using the second refractive power of the second lens part 140.

FIG. 20 is a block diagram of an AR device 800 according to anembodiment.

Referring to FIG. 20, the AR device 800 may include the user input unit150, the memory 160, the processor 170, and a microphone 890, togetherwith an optical system including the optical engine 110, the firstfocus-tunable lens 131, and the second focus-tunable lens 141. Amongcomponents shown in FIG. 20, a component having the same referencenumeral as that of a component shown in FIG. 4 is the same as thecomponent shown in FIG. 4, and thus will not be described redundantly.

The microphone 890 may receive an external audio signal and process thereceived audio signal into electric voice data. For example, themicrophone 890 may receive an audio signal from an external device or aspeaker. The microphone 890 may use various noise cancellationalgorithms for canceling noise generated during reception of theexternal audio signal. The microphone 890 may receive a voice input ofthe user to control the AR device 800. The microphone 890 may receive avoice input of the user who reads a character (602 of FIG. 13) displayedthrough the AR device 800.

Referring to FIGS. 21 through 23, an example of a detailed operation forobtaining a correction-required refractive power of the user will bedescribed.

FIG. 21 illustrates an example where the AR device 800 according to anembodiment of the disclosure performs an operation to obtain acorrection-required refractive power of a user when a correct answerrate of a voice input of the user is low.

Referring to FIG. 21, the AR device 800 may sequentially displaycharacters of a certain size at a focal distance for vision measurement,and receive a voice input of the user with respect to the displayedcharacters. For example, the AR device 800 may sequentially display acharacter B 812, a character O 814, and a character E 816 in differentpositions on a virtual vision measurement board 801 displayed at thefocal distance for vision measurement. In this case, the character B812, the character O 814, and the character E 816 displayed on a visionmeasurement board 801 may be excessively blurredly shown to a userhaving poor vision, as shown in FIG. 21. Thus, the AR device 800 maydisplay the character B 812 and then receive the voice input of theuser, “I can't see it”. Thereafter, the AR device 800 may display thecharacter O 814 and then receive the voice input of the user, “It's 8”.Thereafter, the AR device 800 may also display the character E 816 andthen receive the voice input of the user, “It's 6”.

The AR device 800 may identify the voice input “I can't see it”, comparethe character O with the character 8, and compare the character E withthe character 6. The AR device 800 may also identify a correct answerrate of the voice input of the user as 0% based on comparison results,and change the refractive power of the first focus-tunable lens 131 from0D to −2D.

FIG. 22 illustrates an example where the AR device 800 according to anembodiment of the disclosure performs an operation to obtain acorrection-required refractive power of a user when a correct answerrate of a voice input of the user is normal.

Referring to FIG. 22, the AR device 800 may sequentially displaycharacters of a certain size at a focal distance for vision measurement,and receive a voice input of the user with respect to the displayedcharacters, after the refractive power of the first focus-tunable lens131 changes to ‘−2D’. For example, the AR device 800 may sequentiallydisplay a character B 822, a character E 824, and a character O 826 indifferent positions on a virtual vision measurement board 802 displayedat the focal distance for vision measurement. In this case, the visionmeasurement board 802 may be the same as the vision measurement board801. The character B 822, the character E 824, and the character O 826displayed on the vision measurement board 802 may be moderatelyblurredly shown to the user, as shown in FIG. 22. Thus, the AR device800 may display the character B 822 and then receive the voice input ofthe user, “It's 8”. Thereafter, the AR device 800 may display thecharacter E 824 and then receive the voice input of the user, “It's 6”.Thereafter, the AR device 800 may also display the character O 826 andthen receive the voice input of the user, “It's O”.

The AR device 800 may compare the character B with the voice input 8 andcompare the character E with the voice input O. The AR device 800 mayalso identify the correct answer rate of the voice input of the user as33.3% based on comparison results, and change the refractive power ofthe first focus-tunable lens 131 from −2D to −3D.

FIG. 23 illustrates an example where the AR device 800 according to anembodiment of the disclosure performs an operation to obtain acorrection-required refractive power of a user when a correct answerrate of a voice input of the user is high.

Referring to FIG. 23, the AR device 800 may sequentially displaycharacters of a certain size at a focal distance for vision measurement,and receive a voice input of the user with respect to the displayedcharacters, after the refractive power of the first focus-tunable lens131 changes to ‘−3D’, as shown in FIG. 22. For example, the AR device800 may sequentially display a character B 832, a character O 834, and acharacter E 836 in different positions on a virtual vision measurementboard 803 displayed at the focal distance for vision measurement. Inthis case, the vision measurement board 803 may be the same as thevision measurement board 801. The character B 832, the character O 834,and the character E 836 displayed on the vision measurement board 803may be clearly shown to the user, as shown in FIG. 23. The AR device 800may display the character B 832 and then receive the voice input of theuser, “It's B”. Thereafter, the AR device 800 may also display thecharacter O 834 and then receive the voice input of the user, “It's O”.Thereafter, the AR device 800 may also display the character E 836 andthen receive the voice input of the user, “It's E”.

The AR device 800 may compare the character B with the voice input B,compare the character O with the voice input O, and compare thecharacter E with the voice input E. In addition, the AR device 800 mayalso identify the correct answer rate of the voice input of the user as100% based on comparison results, and convert the correction-requiredrefractive power D_(correction) of the user or the modifiedcorrection-required refractive power D_(modified) from the currentrefractive power (i.e., the first refractive power) of the firstfocus-tunable lens 131, using Equations 11 and 12 provided below.

$\begin{matrix}{D_{correction} = {D_{1C} + D_{fixed} + \frac{1}{f}}} & \left\lbrack {{Equation}\mspace{14mu} 11} \right\rbrack \\{D_{modified} = {{{- D_{fixed}} + D_{correction}} = {D_{1\; C} + \frac{1}{f}}}} & \left\lbrack {{Equation}\mspace{14mu} 12} \right\rbrack\end{matrix}$

D_(1C) indicates the current refractive power of the first focus-tunablelens 131, and f indicates a focal distance of the virtual image and, inthe embodiment, a distance from the eye of the user to the visionmeasurement boards 801, 802, and 803.

The user's correction-required refractive power D_(correction) or themodified correction-required refractive power D_(modified) determined asdescribed above may be stored in the memory 160 to calculate the firstrefractive power of the first focus-tunable lens 131 and the secondrefractive power of the second focus-tunable lens 141.

Although it is described with reference to FIGS. 21 through 23 thatafter three characters are sequentially displayed and correspondingvoice inputs of the user are received, the refractive power of the firstfocus-tunable lens 131 is additionally changed, the number of displayedcharacters is not limited thereto. For example, after the AR device 800may display one character and receive a corresponding voice input of theuser, the AR device 800 may determine whether the user's voice input iscorrect. When the user inputs a wrong answer, the AR device 800 maychange the refractive power of the first focus-tunable lens 131.

The refractive power of the first focus-tunable lens 131 may be changeddifferently from a change level of the refractive power in FIGS. 21 and22. In this case, a change level of the refractive power of the firstfocus-tunable lens 131 may be set variously based on the correct answerrate of the user. For example, for a low correct answer rate of theuser, the AR device 800 may reduce the number of times the refractivepower is changed to correct the user's vision, by changing therefractive power of the first focus-tunable lens 131 many times. Forexample, for a high correct answer rate of the user, the AR device 800may minutely change the user's vision by changing the refractive powerof the first focus-tunable lens 131 a small number of times.

FIG. 24 is a flowchart for describing an operation of the AR device 800according to an embodiment.

Referring to FIG. 24, for an operation of obtaining thecorrection-required refractive power of the user, at least one firstcharacter of a preset size may be output on the virtual visionmeasurement board 801 displayed at a focal distance for visionmeasurement through the optical engine 110 and at least one first voiceinput of the user with respect to the at least one first character maybe obtained, in operation S910. Next, the at least one first characterand the at least one first voice input may be compared with each other,in operation S920. Next, the first refractive power of the firstfocus-tunable lens 131 may be determined based on a comparison result,in operation S930. As in Equation 11 described above, the user'scorrection-required refractive power D_(correction) may be determinedbased on the determined first refractive power of the firstfocus-tunable lens 131, in operation S940. As in Equation 12 describedabove, the modified correction-required refractive power D_(modified)may be determined based on the determined first refractive power of thefirst focus-tunable lens 131.

While the voice input of the user is received through the microphone 890in the embodiment described with reference to FIGS. 20 through 24,embodiments are not limited thereto. For example, the AR device 800 mayreceive information about user's reading of a character through a user'stouch input, etc., with the user input interface 150.

An embodiment may be implemented using a recording medium including acomputer-executable command such as a computer-executable programmingmodule. A computer-readable recording medium may be an available mediumthat is accessible by a computer, and includes all of a volatile medium,a non-volatile medium, a separated medium, and a non-separated medium.The computer-readable recording medium may also include a computerstorage medium and a communication medium. The computer storage mediumincludes all of a volatile medium, a non-volatile medium, a separatedmedium, and a non-separated medium, which is implemented by a method ortechnique for storing information such as a computer-readableinstruction, a data structure, a programming module, or other data. Acommunication medium may typically include a computer-readableinstruction, a data structure, or other data of a modulated data signalsuch as a programming module.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. Wherein, the term “non-transitory” simplymeans that the storage medium is a tangible device, and does not includea signal (e.g., an electromagnetic wave), but this term does notdifferentiate between where data is semi-permanently stored in thestorage medium and where the data is temporarily stored in the storagemedium. For example, the ‘non-transitory storage medium’ may include abuffer in which data is temporarily stored.

According to an embodiment of the disclosure, a method according tovarious embodiments of the disclosure may be included and provided in acomputer program product. The computer program product may be traded asa product between a seller and a buyer. The computer program product maybe distributed in the form of a machine-readable storage medium (e.g.,compact disc read only memory (CD-ROM)), or be distributed (e.g.,downloaded or uploaded) online via an application store (e.g., PlayStore™), or between two user devices (e.g., smart phones) directly. Whendistributed online, at least a part of the computer program product(e.g., a downloadable app) may be temporarily generated or at leasttemporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

In the specification, the term “unit” may be a hardware component like aprocessor or a circuit, and/or a software component executed by ahardware component like a processor.

Those of ordinary skill in the art to which the disclosure pertains willappreciate that the disclosure may be implemented in different detailedways without departing from the technical spirit or essentialcharacteristics of the disclosure. Accordingly, the aforementionedembodiments of the disclosure should be construed as being onlyillustrative, but should not be constructed as being restrictive fromall aspects. For example, each element described as a single type may beimplemented in a distributed manner, and likewise, elements described asbeing distributed may be implemented as a coupled type.

According to embodiments of the disclosure, a device and method ofdisplaying AR may provide a self-vision correction function.

According to embodiments of the disclosure, a device and method ofdisplaying AR may provide an immersive AR environment by moving a focaldistance of a virtual image to a random position where a real object islocated.

According to embodiments the disclosure, a device and method ofdisplaying AR may improve the qualities of the virtual image and a realscene by reducing a refractive power required level of a focus-tunablelens.

While the device and method of displaying AR according to embodiments ofthe disclosure has been shown and described in connection with theembodiments to help understanding of the disclosure, it will be apparentto those of ordinary skill in the art that modifications and variationsmay be made. Therefore, the true technical scope of the disclosureshould be defined by the appended claims and their equivalents.

What is claimed is:
 1. A device for displaying augmented reality (AR),the device comprising: an optical engine configured to output light of avirtual image; a waveguide configured to output the light of the virtualimage received from the optical engine and transmit light of a realscene; a first lens part provided on a first surface of the waveguide; asecond lens part provided on a second surface of the waveguide oppositeto the first surface; and a processor, wherein the first lens part isconfigured to tune a focus of the virtual image and correct a user'svision, the first lens part comprising a first focus-tunable lens havinga first refractive power that is tunable by the processor and a fixedrefractive lens having a fixed refractive power, wherein the second lenspart is configured to compensate distortion of the real scene caused bythe first lens part, and the second lens part comprising a secondfocus-tunable lens having a second refractive power that is tunable bythe processor, and wherein the processor is further configured todetermine the first refractive power of the first focus-tunable lensbased on vision information of the user, attribute depth information ofthe virtual image, and fixed refractive power information of the fixedrefractive lens.
 2. The device of claim 1, wherein the first refractivepower of the first focus-tunable lens satisfies${D_{1} = {{- D_{fixed}} + D_{correction} - \frac{1}{f}}},$ where D₁indicates the first refractive power of the first focus-tunable lens,D_(fixed) indicates the fixed refractive power of the fixed refractivelens, D_(correction) indicates a correction-required refractive powerfor correcting ametropia of the user, and f indicates a focal distanceof the virtual image.
 3. The device of claim 2, further comprising amemory configured to store the fixed refractive power D_(fixed) of thefixed refractive lens, the correction-required refractive powerD_(correction) of the user, and the focal distance f of the virtualimage, wherein the processor is further configured to read the fixedrefractive power of the fixed refractive lens, the correction-requiredrefractive power of the user, and focal distance of the virtual imagefrom the memory and obtain the first refractive power D₁ of the firstfocus-tunable lens as ${- D_{fixed}} + D_{correction} - {\frac{1}{f}.}$4. The device of claim 2, further comprising a memory configured tostore the focal distance f of the virtual image and a modifiedcorrection-required refractive power D_(modified) in which the fixedrefractive power of the fixed refractive lens is reflected, wherein themodified correction-required refractive power D_(modified) satisfiesD_(modified)=−D_(fixed)+D_(correction), and wherein the processor isfurther configured to read the modified correction-required refractivepower and the focal distance of the virtual image from the memory andobtain the first refractive power D₁ of the first focus-tunable lens as$D_{modified} - {\frac{1}{f}.}$
 5. The device of claim 1, wherein asecond refractive power D₂ of the second focus-tunable lens satisfies$D_{2} = {\frac{1}{f}.}$
 6. The device of claim 1, wherein the fixedrefractive lens is a concave lens having a negative (−) refractivepower.
 7. The device of claim 1, wherein the first focus-tunable lensand the second focus-tunable lens are liquid crystal lenses.
 8. Thedevice of claim 1, wherein the second focus-tunable lens is providedbetween the waveguide and the fixed refractive lens, and wherein thefirst focus-tunable lens, the waveguide, and the second focus-tunablelens have a stack structure.
 9. The device of claim 1, furthercomprising a user input interface configured to receive at least any oneof the vision information of the user or the focal distance of thevirtual image based on a user input.
 10. The device of claim 1, whereinthe first lens part further comprises a polarization plate provided onan incident surface of the fixed refractive lens or an emission surfaceof the fixed refractive lens.
 11. The device of claim 1, wherein thesecond lens part further comprises a second fixed refractive lensconfigured to compensate distortion of the real scene caused by thefirst lens part and the second focus-tunable lens.
 12. The device ofclaim 11, wherein the second fixed refractive lens is a convex lenshaving a positive (+) refractive power.
 13. The device of claim 11,wherein the second refractive power D₂ of the second focus-tunable lenssatisfies ${D_{2} = {\frac{1}{f} - D_{{fixed}\; 2}}},$ where D_(fixed2)indicates a fixed refractive power of the second fixed refractive lens.14. The device of claim 1, further comprising a gaze tracking sensorconfigured to obtain gaze information of the user.
 15. The device ofclaim 14, wherein the processor is further configured to: obtain a gazepoint from the gaze information of the user obtained by the gazetracking sensor; and determine the focal distance of the virtual imagebased on the obtained gaze point.
 16. The device of claim 1, wherein theprocessor is further configured to: control the optical engine to outputat least one first character of a preset size; obtain at least one firstuser input with respect to the at least one first character; compare theat least one first character with the at least one first user input;determine the first refractive power of the first focus-tunable lensbased on a result of the comparing; and determine thecorrection-required refractive power of the user based on the determinedfirst refractive power of the first focus-tunable lens.
 17. The deviceof claim 16, wherein the at least one first character and at least onesecond character have sizes corresponding to preset corrected vision,and wherein the at least one first character and the at least one secondcharacter are displayed to a preset depth for vision measurement of theuser.
 18. The device of claim 1, wherein the device is a glasses-typedevice.
 19. A method of displaying augmented reality (AR) in an ARdevice that comprises an optical engine configured to output light of avirtual image and a waveguide configured to output the light of thevirtual image and transmit light of a real scene, the method comprising:providing a first lens part comprising a fixed refractive lens and afirst focus-tunable lens and a second lens part comprising a secondfocus-tunable lens on opposite surfaces of the waveguide; obtaining afirst refractive power of the first focus-tunable lens based on visioninformation of a user, focal distance of the virtual image, and a fixedrefractive power of the fixed refractive lens; and obtaining a secondrefractive power of the second focus-tunable lens to compensate fordistortion of the real scene caused by the first lens part.
 20. Themethod of claim 19, wherein the obtaining of the first refractive powerof the first focus-tunable lens comprises: reading a fixed refractivepower D_(fixed) of the fixed refractive lens, a correction-requiredrefractive power of the user, and the focal distance f of the virtualimage from a memory; and obtaining a first refractive power D₁ of thefirst focus-tunable lens satisfying${{D1} = {{- D_{fixed}} + D_{correction} - \frac{1}{f}}}.$